Millimeter and terahertz wave photodetectors have a wide range of applications. However, the state-of-the-art techniques lag far behind the urgent demand due to the structure and performance limitations. Here, we report sensitive and direct millimeter and terahertz wave photodetection in compact InGaAs-based subwavelength ohmic metal–semiconductor–metal structures. The photoresponse originates from unidirectional transportation of nonequilibrium electrons induced by surface plasmon polaritons under irradiation. The detected quantum energies of electromagnetic waves are far below the bandgap of InGaAs, offering, to the best of our knowledge, a novel direct photoelectric conversion pathway for InGaAs beyond its bandgap limit. The achieved room temperature rise time and noise equivalent power of the detector are 45 μs and 20 pW·Hz 1/2, respectively, at the 0.0375 THz (8 mm) wave. The detected wavelength is tunable by mounting different coupling antennas. Room temperature terahertz imaging of macroscopic samples at around 0.166 THz is also demonstrated. This work opens an avenue for sensitive and large-area uncooled millimeter and terahertz focal planar arrays.